1. Field of the Invention
The present invention relates to an image processing apparatus, more particularly, to an image processing apparatus which detects and corrects a defect, and control method thereof.
2. Description of the Related Art
In a recent digital image processing technique, the progress in a field of study called light field photography is significant. In light field photography, first, an image is captured so that a captured image obtained from an image capturing apparatus including an imaging optical system such as a lens, and a solid-state imaging sensor such as a CCD or CMOS sensor includes not only two-dimensional light intensity information of the object field, but also light beam angle information.
The captured image including the light beam angle information is equivalent to division into a considerable number of pupils exceeding division into a pair of pupils in so-called phase difference autofocus detection (AF). This captured image is data not always having the list and order of pieces of significant information when the image is directly observed. To solve this, there have been proposed an image capturing apparatus and image processing apparatus capable of focusing again (to be also referred to as refocusing) on an arbitrary object in the object field after imaging by performing reconstruction image processing closely related to an image capturing process in which the captured image was obtained.
An example of this image capturing apparatus and image processing apparatus is a hand-held plenoptic camera (to be also referred to as a light field camera hereinafter) described in literature 1 “Ren. Ng, et al., ‘Light Field Photography with a Hand-Held Plenoptic Camera’, Stanford Tech Report CTSR 2005-02”. The building components of the light field camera are as follows: a “main lens”, which is an imaging optical system in a well-known image capturing apparatus, forms mainly an object image on a “microlens” array having a predetermined pitch. A “solid-state imaging sensor” including photoelectric conversion elements at a smaller pitch than the predetermined pitch is arranged behind the microlens array.
In other words, by a special image capturing process and image processing premised on this, the light field camera described in literature 1 obtains an image having new information which has not been obtained by a well-known image capturing apparatus.
To the contrary, most image processes which are performed in a conventionally well-known image capturing apparatus are executed on the assumption that the signals of photoelectric conversion elements, which constitute a captured image, have continuity to some extent. An example is a real-time defective pixel detection method of determining a defective pixel from a level difference from the signal of a surrounding photoelectric conversion element in every imaging by the image capturing apparatus.
For example, Japanese Patent Laid-Open No. 2005-286825 discloses a defective pixel correction apparatus characterized by comprising a first defective pixel detection unit configured to detect a defective pixel by comparing, with a predetermined value, each of signals output from a plurality of pixels, a first correction unit configured to correct a signal from the defective pixel detected by the first defective pixel detection unit, a second defective pixel detection unit configured to detect a defective pixel by comparing, with a predetermined value, each of signals output from the plurality of pixels in which correction has been performed on the defective pixel detected by the first defective pixel detection unit, and a setting control unit configured to set the predetermined values to be different from each other in the first defective pixel detection unit and the second defective pixel detection unit.
Japanese Patent Laid-Open No. 2005-286825 also discloses that information of a defective pixel detected by the first defective pixel detection unit is recorded, and the second defective pixel detection unit refers to even the defective pixel information in the first defective pixel detection unit. Further, Japanese Patent Laid-Open No. 2005-286825 describes that it is determined whether the detected pixel is actually a defective pixel or an object edge, and if the detected pixel is considered to be an object edge, no correction is performed to prevent degradation of the image quality.
Real-time defective pixel detection is especially effective to suppress degradation of the image quality accompanying a subsequent defective pixel which has not been recorded in the memory of the image capturing apparatus or the like, or a flickering defective pixel whose output level varies in every imaging. However, the most important task in the real-time defective pixel detection method is discrimination between a defective pixel which should be a correction target, and an object edge.
In a well-known image capturing apparatus, a unit element constituting a captured image is generally called a “pixel”. It has not been necessary to definitely discriminate the “signal of a photoelectric conversion element” serving as a unit element constituting a solid-state imaging sensor, and the “pixel” serving as a unit element constituting a final image having undergone signal processing. In the light field camera, however, they are concepts which should be definitely discriminated.
As described above, to obtain a significant feature such as refocus reconstruction processing, a special image capturing process as mentioned above occupies an important position in the light field camera. The special image capturing process means simultaneous acquisition of light intensity information and light beam angle information. The light beam angle information appears in the intensity distribution of many photoelectric conversion elements corresponding to one microlens. At this time, there is an image shift in which the intensity distribution of the signals of a plurality of photoelectric conversion elements shifts in accordance with the distance to an object and focusing of the imaging optical system. The image shift is a phenomenon in which a coordinate shift occurs in images constituted by the signals of photoelectric conversion elements belonging to the same quadrant of microlenses. The image shift is regarded as expansion of a shift between a pair of images to a shift between many images in phase difference detection AF.
More specifically, a captured image obtained by arranging the signals of many photoelectric conversion elements in the output order of the solid-state imaging sensor without performing any refocus reconstruction processing is data not suitable for direct observation. It is therefore difficult to accomplish the task of discriminating an object edge and a defective pixel by only applying, to a captured image including light beam angle information, the aforementioned real-time defective pixel detection method which assumes that the signals of photoelectric conversion elements have continuity to some extent. In the light field camera, the address of a pixel in a reconstructed image that may be influenced by the signal of a defective photoelectric conversion element (to be also referred to as a defective element hereinafter) changes upon every refocusing in refocus reconstruction processing.
For this reason, to suppress degradation of the image quality accompanying a defective element in the light field camera, it is necessary to perform real-time defective pixel detection on a reconstructed image upon every refocus reconstruction processing, and further correct the detected defective pixel. However, real-time defective pixel detection takes the processing time to scan a reconstructed image. This means that much time is taken until a reconstructed image is displayed after an operation of focusing again on an arbitrary object in the object field of a captured image. The user may feel uncomfortable.